Proposal Image - based Capture and Modeling of Dynamic Human Motion and Appearance Name of Student : Neil Birkbeck

The goal of the proposed research is to capture a concise representation of human deformation and appearance in a low-cost environment that can be seamlessly integrated into entertainment and Virtual Reality applications–applications that often require relighting, high levels of realism, and rendering from novel viewpoints. Techniques for modeling geometric deformation of humans exist in the current state of the field, but they focus solely on geometric deformation and ignore the appearance component. I propose a combined model of both geometric and appearance deformation, leading to a more compact representation while still achieving high levels of realism. The proposed vision-based method is less expensive than, e.g., Laser scanners or motion capture, and requires only limited manual intervention. In a controlled environment, several cameras will be used to acquire a dense static geometry and a basic appearance model of a street-clothed human (e.g., a texture mapped model), and skeletal model will be aligned. The actor performs a predefined sequence of motion, from which multiimage vision-based tracking techniques are used to extract the gross kinematic motion parameters, and multi-view stereo, silhouette, and temporal cues are used to extract the time-varying residual geometric deformation. These time-dependent geometric quantities, and the time and view-dependent photometric quantities are used to create the final compact model of appearance and geometric variation. Following several graphics techniques for modeling articulated deformation and leveraging existing tools for image-based rendering, the deformation model is built on top of linear blend skinning. Appearance and geometric variation are modeled together as a function of abstract interpolating parameters that include, e.g., relative joint angles and view angles. Examples of deformation include muscle bulging and some forms of cloth budging. We model this compactly using a dimensionality reduction and sparse data interpolation, where several low-dimensional subspaces and corresponding interpolation spaces are used, clustering the surface into regions that are affected by the same abstract interpolators. Contrast to many of the vision-based human modeling techniques, our complete model can be re-rendered under novel viewpoint, novel animation, and even novel illumination when the illumination during capture is calibrated. Potential applications include appearance and deformation transfer between subjects, and usage of the model in related domains, e.g., to model facial deformations.